Enhance Interval arithmetic to support String type.  #7098

Activity

1. mingmwang

   added

   enhancementNew feature or requestNew feature or request

   on Jul 26, 2023
2. mingmwang commented on Jul 26, 2023

   

   mingmwang

   on Jul 26, 2023

   ContributorAuthor

   More actions

   @berkaysynnada @ozankabak
   Please share your thoughts on this.
3. mingmwang commented on Jul 26, 2023

   

   mingmwang

   on Jul 26, 2023

   Last edited by mingmwang

   ContributorAuthor

   More actions

       #[test]
       fn test_normalize_string_interval() -> Result<()> {
           // Define a mapping of characters to numerical values
           let mut char_map: HashMap<char, u8> = HashMap::new();
   
           for i in 0..128 {
               let c = (i as u8) as char;
               char_map.insert(c, i as u8);
           }
   
           let mut string_intervals = vec!["00", "aaaaa", "AMddist", "!_xyz", "zzz", "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzx", "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"];
           // sort the string_intervals by their ascii value
           string_intervals.sort_by(|a, b| a.cmp(b));
   
           let first_interval = string_intervals.first().unwrap();
           let last_interval = string_intervals.last().unwrap();
   
           let float_vec = string_intervals.iter().map(|interval| {
               calcuate_normalize_value(&mut char_map, interval).unwrap_or(0f64)
           }).collect::<Vec<_>>();
   
           let min_val = float_vec.iter().fold(std::f64::INFINITY, |acc, x| acc.min(*x));
           let max_val = float_vec.iter().fold(std::f64::NEG_INFINITY, |acc, x| acc.max(*x));
           let normalized_vector: Vec<f64> = float_vec.iter().map(|x| (x - min_val) / (max_val - min_val)).collect();
   
           println!("normalized_vector {:?}", normalized_vector);
           Ok(())
       }
   
       fn calcuate_normalize_value(char_map: &HashMap<char, u8>, str_value: &str) -> Option<f64> {
           let base = char_map.len() as f64;
           if str_value.is_ascii() {
               let mut sum = 0.0;
               for (i, c) in str_value.chars().enumerate() {
                   let char_value = *char_map.get(&c).unwrap() as f64;
                   let weight = base.powf((0-i as i32) as f64);
                   sum += char_value * weight;
               }
               let normalized_value = sum / char_map.len() as f64;
               Some(normalized_value)
           } else {
               None
           }
       }
4. berkaysynnada commented on Jul 26, 2023

   

   berkaysynnada

   on Jul 26, 2023

   Contributor

   More actions

   Sounds good. However, I'm not entirely sure about the correctness of the weight calculation. Shouldn't the weight of the leading characters be higher (assuming the order of the strings is as in the dictionary)? Once the strings are accurately mapped to floats, I believe there wouldn't be any problem with calculating selectivity as you've proposed.
5. mingmwang commented on Jul 26, 2023

   

   mingmwang

   on Jul 26, 2023

   ContributorAuthor

   More actions

   This will output:

   normalized_vector [0.0, 0.16394189376465151, 0.35710980460519687, 0.7175591189901976, 0.9999993427698617, 1.0, 1.0]

   👍React with 👍1Reacted by Berkay Şahin
6. mingmwang commented on Jul 26, 2023

   

   mingmwang

   on Jul 26, 2023

   ContributorAuthor

   More actions

   Sounds good. However, I'm not entirely sure about the correctness of the weight calculation. Shouldn't the weight of the leading characters be higher (assuming the order of the strings is as in the dictionary)? Once the strings are accurately mapped to floats, I believe there wouldn't be any problem with calculating selectivity as you've proposed.

   Yes, you are right. The weigh is calculated based on the position of the char.
7. berkaysynnada commented on Jul 26, 2023

   

   berkaysynnada

   on Jul 26, 2023

   Contributor

   More actions

   This version seems correct to calculate weights. I don't think there will be any problems and it would improve the interval arithmetic and statistics calculations. Thanks
8. alamb commented on Oct 20, 2023

   

   alamb

   on Oct 20, 2023

   Contributor

   More actions

   Started #7882 to collect improvement to interval arithmetic